Video representation using a sparsity-based model

ABSTRACT

A method for representing a video sequence including a time sequence of input video frames, the input video frames including some common scene content that is common to all of the input video frames and some dynamic scene content that changes between at least some of the input video frames. Affine transform are determined to align the common scene content in the input video frames. A common video frame including the common scene content is determined by forming a sparse combination of a first basis functions. A dynamic video frame is determined for each input video frame by forming a sparse combination of a second basis functions, wherein the dynamic video frames can be combined with the respective affine transforms and the common video frame to provide reconstructed video frames.

CROSS-REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned, co-pending U.S. patentapplication Ser. No. ______ (Docket K000878), entitled: “Scene boundarydetermination using sparsity-based model”, by Kumar et al., which isincorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to the field of video understanding,and more particularly to a method to represent the static and thedynamic content of the video using a sparse representation.

BACKGROUND OF THE INVENTION

With the development of digital imaging and storage technologies, videoclips can be conveniently captured by consumers using various devicessuch as camcorders, digital cameras or cell phones and stored for laterviewing and processing. Efficient content-aware video representationmodels are critical for many video analysis and processing applicationsincluding denoising, restoration, and semantic analysis.

Developing models to capture spatiotemporal information present in videodata is an active research area and several approaches to representvideo data content effectively have been proposed. For example, Cheunget al. in the article “Video epitomes” (Proc. IEEE Conference onComputer Vision and Pattern Recognition, Vol. 1, pp. 42-49, 2005), teachusing patch-based probability models to represent video content.However, their model does not capture spatial correlation.

In the article “Recursive estimation of generative models of video”(Proc. IEEE Conference on Computer Vision and Pattern Recognition, Vol.1, pp. 79-86, 2006), Petrovic et al. teach a generative model andlearning procedure for unsupervised video clustering into scenes.However, they assume videos to have only one scene. Furthermore, theirframework does not model local motion.

Peng et al., in the article “RASL: Robust alignment by sparse andlow-rank decomposition for linearly correlated images” (Proc. IEEEConference on Computer Vision and Pattern Recognition, pp. 763-770,2010), teach a sparsity-based method for simultaneously aligning a batchof linearly correlated images. Clearly, this model is not suitable forvideo processing as video frames, in general, are not linearlycorrelated.

Another method taught by Baron et al., in the article “Distributedcompressive sensing” (preprint, 2005), models both intra- andinter-signal correlation structures for distributed coding algorithms.

In the article “Compressive acquisition of dynamic scenes” (Proc. 11thEuropean Conference on Computer Vision, pp. 129-142, 2010),Sankaranarayanan et al. teach a compressed sensing-based model forcapturing video data at much lower rate than the Nyquist frequency.However, this model works only for single scene video.

In the article “A compressive sensing approach for expression-invariantface recognition” (Proc. IEEE Conference on Computer Vision and PatternRecognition, pp. 1518-1525, 2009), Nagesh et al. teaches a facerecognition algorithm based on the theory of compressed sensing. Given aset of registered training face images from one person, their algorithmestimates a common image and a series of innovation images. Theinnovation images are further exploited for face recognition. However,this algorithm is not suitable for video modeling as it was designedexplicitly for face recognition and does not preserve pixel-levelinformation.

There remains a need for a video representation framework that is dataadaptive, robust to noise and different content, and can be applied towide varieties of videos including reconstruction, denoising, andsemantic understanding.

SUMMARY OF THE INVENTION

The present invention represents a method for representing a videosequence including a time sequence of input video frames, the inputvideo frames including some common scene content that is common to allof the input video frames and some dynamic scene content that changesbetween at least some of the input video frames, comprising:

a) determining an affine transform having a set of affine transformcoefficients for each input video frame that can be used to align thecommon scene content in the input video frames;

b) defining a first set of basis functions for representing the commonscene content;

c) defining a second set of basis functions for representing the dynamicscene content;

d) automatically determining a common video frame including the commonscene content by forming a sparse combination of the first basisfunctions;

e) automatically determining a dynamic video frame for each input videoframe by forming a sparse combination of the second basis functions,wherein the dynamic video frames can be combined with the respectiveaffine transforms and the common video frame to provide reconstructedvideo frames providing representations of the respective input videoframes; and

f) storing representations of the common video frame and the dynamicvideo frames in a processor-accessible memory;

wherein the method is performed at least in part using a data processingsystem.

The present invention has the advantage that it can detect local motionpresent in the scene without performing any motion estimation.

It has the additional advantage that it can be used for complex videoprocessing tasks such as reconstruction, denoising, and semanticunderstanding.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram showing the components of a system forsummarizing digital video according to an embodiment of the presentinvention;

FIG. 2 is a flow diagram illustrating a method for determining commonand dynamic scene contents from a video sequence according to anembodiment of the present invention;

FIG. 3 is a block diagram showing a detailed view of the get affinetransform coefficients step of FIG. 2;

FIG. 4 is a block diagram showing a detailed view of the get common anddynamic video frames step of FIG. 2;

FIG. 5 shows example common and dynamic scene content results obtainedaccording to an embodiment of the present invention;

FIG. 6 is a flow diagram illustrating a method for reconstructing avideo segment from its common and dynamic scene content according to anembodiment of the present invention;

FIG. 7 shows example denoising results obtained according to anembodiment of the present invention;

FIG. 8 is a flow diagram illustrating a method for changing the commonscene content of a video segment according to an embodiment of thepresent invention;

FIG. 9 is a flow diagram illustrating a method for tracking movingobjects according to an embodiment of the present invention;

FIG. 10 is a flow diagram illustrating a method for determining a sceneboundary between a first scene and a second scene in an input videosequence according to an embodiment of the present invention;

FIG. 11 is a diagram showing the extraction of overlapping digital videosections from a digital video according to an embodiment of the presentinvention;

FIG. 12A is a graph plotting a merit function value as a function ofcandidate scene boundary location for a digital video section includinga scene boundary;

FIG. 12B is a graph plotting a merit function value as a function ofcandidate scene boundary location for a digital video section that doesnot include a scene boundary; and

FIG. 13 is a flow diagram illustrating a method for computing the meritfunction values of FIG. 10 according to an embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to the “method” or “methods” and thelike is not limiting.

The phrase, “digital content record”, as used herein, refers to anydigital content record, such as a digital still image, a digital audiofile, or a digital video file.

It should be noted that, unless otherwise explicitly noted or requiredby context, the word “or” is used in this disclosure in a non-exclusivesense.

FIG. 1 is a high-level diagram showing the components of a system forprocessing a digital video sequence according to an embodiment of thepresent invention. The system includes a data processing system 110, aperipheral system 120, a user interface system 130, and a data storagesystem 140. The peripheral system 120, the user interface system 130 andthe data storage system 140 are communicatively connected to the dataprocessing system 110.

The data processing system 110 includes one or more data processingdevices that implement the processes of the various embodiments of thepresent invention, including the example processes of FIGS. 2-11described herein. The phrases “data processing device” or “dataprocessor” are intended to include any data processing device, such as acentral processing unit (“CPU”), a desktop computer, a laptop computer,a mainframe computer, a personal digital assistant, a Blackberry™, adigital camera, cellular phone, or any other device for processing data,managing data, or handling data, whether implemented with electrical,magnetic, optical, biological components, or otherwise.

The data storage system 140 includes one or more processor-accessiblememories configured to store information, including the informationneeded to execute the processes of the various embodiments of thepresent invention, including the example processes of FIGS. 2-11described herein. The data storage system 140 may be a distributedprocessor-accessible memory system including multipleprocessor-accessible memories communicatively connected to the dataprocessing system 110 via a plurality of computers or devices. On theother hand, the data storage system 140 need not be a distributedprocessor-accessible memory system and, consequently, may include one ormore processor-accessible memories located within a single dataprocessor or device.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, floppy disks, hard disks, Compact Discs,DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data may be communicated.

The phrase “communicatively connected” is intended to include aconnection between devices or programs within a single data processor, aconnection between devices or programs located in different dataprocessors, and a connection between devices not located in dataprocessors at all. In this regard, although the data storage system 140is shown separately from the data processing system 110, one skilled inthe art will appreciate that the data storage system 140 may be storedcompletely or partially within the data processing system 110. Furtherin this regard, although the peripheral system 120 and the userinterface system 130 are shown separately from the data processingsystem 110, one skilled in the art will appreciate that one or both ofsuch systems may be stored completely or partially within the dataprocessing system 110.

The peripheral system 120 may include one or more devices configured toprovide digital content records to the data processing system 110. Forexample, the peripheral system 120 may include digital still cameras,digital video cameras, cellular phones, or other data processors. Thedata processing system 110, upon receipt of digital content records froma device in the peripheral system 120, may store such digital contentrecords in the data storage system 140.

The user interface system 130 may include a mouse, a keyboard, anothercomputer, or any device or combination of devices from which data isinput to the data processing system 110. In this regard, although theperipheral system 120 is shown separately from the user interface system130, the peripheral system 120 may be included as part of the userinterface system 130.

The user interface system 130 also may include a display device, aprocessor-accessible memory, or any device or combination of devices towhich data is output by the data processing system 110. In this regard,if the user interface system 130 includes a processor-accessible memory,such memory may be part of the data storage system 140 even though theuser interface system 130 and the data storage system 140 are shownseparately in FIG. 1.

FIG. 2 is a flow diagram illustrating a method for representing commonand dynamic scene content of a video according to an embodiment of thepresent invention. An input digital video 203 representing a videosequence captured of a scene is received in a receive input digitalvideo step 202. The video sequence includes a time sequence of videoframes. Each video frame includes an array of pixels having associatedpixel values. The input digital video 203 can be captured using any typeof video capture device known in the art such as a video camera, adigital still camera with a video capture mode or a camera phone, andcan be received in any digital video format known in the art.

An initialize intermediate digital video step 204 is used to initializean intermediate digital video 205. The intermediate digital video 205 isa modified video estimated from the input digital video 203.

A get video segments step 206 detects the scene boundaries (i.e., thescene change locations) in the intermediate digital video 205. Theintermediate digital video 205 is divided at the scene change locationsto provide a set of video segments, which are collected in a videosegments set 207.

A select video segment step 208 selects a particular video segment fromthe video segments set 207 to provide a selected video segment 209.

A get affine transform coefficients step 210 determines an affinetransform having a set of affine transform coefficients for each inputvideo frame of the selected video segment 209. The sets of affinetransform coefficients for each video frame are collected in an affinetransform coefficients set 211. The affine transform coefficients of thevideo frames corresponding to the selected video segment 209 are used toalign the common scene content present in the selected video segment209.

Finally, a get common and dynamic video frames step 212 uses theselected video segment 209 and the affine transform coefficients set 211to determine a common frame and a set of dynamic frames. The commonvideo frame represents the common scene content that is common to all ofthe video frames of the selected video segment 209. The set of dynamicvideo frames represent the scene content that changes between at leastsome of the video frames of the selected video segment 209. The commonvideo frame and dynamic video frames are collected in a common anddynamic video frames set 213.

The individual steps outlined in FIG. 2 will now be described in greaterdetail. The initialize intermediate digital video step 204 is apreprocessing step that preprocesses the input digital video 203 toproduce the intermediate digital video 205. The intermediate digitalvideo 205 is more suitable for the subsequent steps carried out toproduce the common and dynamic video frames set 213. For example, insome embodiments the input digital video 203 is down-sampled to a lowerspatial resolution to provide the intermediate digital video 205.Similarly, the input digital video 203 can be down-sampled temporallysuch that the intermediate digital video 205 has fewer video frames thatneed to be analyzed. In other embodiments, the initialize intermediatedigital video step 204 can apply other types of operations such as tonescale and color adjustments, noise reduction or sharpening operations.

The get video segments step 206 analyzes the intermediate digital video205 to provide the video segments set 207. The video segments set 207represents the scene boundary locations in the intermediate digitalvideo 205. Mathematical algorithms for determining scene boundarylocations are well-known in the art. Any such method can be used inaccordance with the present invention. In a preferred embodiment, theget video segments step 206 uses the method for determining sceneboundary locations that will be described below with respect to FIGS. 10and 11.

The select video segment step 208 selects a video segment from the videosegments set 207 to provide the selected video segment 209. The selectedvideo segment 209 can be selected in any appropriate way known to thoseskilled in the art. In a preferred embodiment, a user interface isprovided enabling a user to manually select the video segment to bedesignated as the selected video segment 209. In other embodiments, thevideo segments set 207 can be automatically analyzed to designate theselected video segment 209 according to a predefined criterion. Forexample, the video segment depicting the maximum amount of local motioncan be designated as the selected video segment 209.

The get affine transform coefficients step 210 determines an affinetransform defined by a set of affine transform coefficients for eachvideo frame of the selected video segment 209. Let T(Θ_(i)) be theaffine transform having the set of affine transform coefficients Θ_(i)corresponding to the i^(th) video frame of the selected video segment209, where 1≦i≦n. The affine transform coefficients Θ_(i) includeparameters for displacement along x- and y-axis, rotation and scalingfor the i^(th) video frame of the selected video segment 209. In apreferred embodiment of the present invention, Θ_(i) contains only thedisplacements along the x- and y-axis (i.e., Θ_(i)={x_(i), y_(i)}, wherex_(i), and y_(i) are global displacements along x- and y-axis,respectively) for the i^(th) video frame of the selected video segment209. The affine transform T(Θ_(i)) is a spatial transform that can beapplied to a given input image z(p, q) to provide a transformed imagez(p′, q′). Functionally this can be expressed as T(Θ_(i))z(p, q)=z(p′,q′), where

$\begin{matrix}{\begin{bmatrix}p^{\prime} \\q^{\prime}\end{bmatrix} = {\begin{bmatrix}p \\q\end{bmatrix} + \begin{bmatrix}x_{i} \\y_{i}\end{bmatrix}}} & (1)\end{matrix}$

The affine transform coefficients Θ_(i) (1≦i≦n) are collected in theaffine transform coefficients set 211. The estimation of Θ_(i) isexplained next.

FIG. 3 is a more detailed view of the get affine transform coefficientsstep 210 according to a preferred embodiment. In a determine transformcoefficients model step 302, a transform model to represent transformcoefficient in affine transform coefficients set 211 is determined. Thetransform model to relate the transform coefficients of video frames canbe determined in any appropriate way known to those skilled in the art.In a preferred embodiment of the present invention, the transformcoefficients model set 303 is represented using an auto regressive modelas given by Eqs. (2) and (3) below:

x _(i) =x _(i−1) +Δx _(i−1)   (2)

and

y _(i) =y _(i−1) +Δy _(i−1)   (3)

where 1≦i≦n. Furthermore, it is assumed that Δx₀=Δy₀=0.

In a determine measurement vector step 304, a set of measurement vectorsis determined responsive to the selected video segment 209. Thedetermined measurement vectors are collected in a measurement vector set305. In the preferred embodiment, the determine measurement vector step304 computes the global displacements in x- and y-directions betweensuccessive video frames of the selected video segment 209. Mathematicalalgorithms for determining global displacements between pair of imagesare well-known in the art. An in-depth analysis of image alignment, itsmathematical structure and relevancy can be found in the article byBrown entitled “A survey of image registration techniques” (ACMComputing Surveys, Vol. 24, issue 4, pp. 325-376, 1992), which isincorporated herein by reference.

An estimate affine transform coefficients step 306 uses the measurementvector set 305 and transform coefficients model set 303 to determine theaffine transform coefficients set 211. The affine transform coefficientsset 211 can be determined in any appropriate way known to those skilledin the art. In a preferred embodiment, the affine transform coefficientsset 211 is determined using a sparse representation framework where themeasurement vector set 305 and the auto regressive model of thetransform coefficients model set 303 are related using a sparse linearrelationship. The affine transform coefficients set 211 is thendetermined responsive to the sparse linear relationship as explainednext.

Let f₁, f₂, . . . , f_(n) be the video frames of the selected videosegment 209. Furthermore, let X=[X₁, X₂, . . . , X_(n-1)]^(T), andY=[Y₁, Y₂, . . . , Y_(n-1)]^(T) be the elements of the measurementvector set 305 corresponding to the selected video segment 209representing global displacements along x- and y-axis, respectively. Thei^(th) (1≦i≦n-1) element of X represents the global displacement betweenvideo frames f_(i) and f_(i+1) in x-direction. Similarly, i^(th) elementof Y represents the global displacement between video frames f_(i) andf_(i+1) in y-direction. In equation form, the sparse linear relationshipbetween X and the auto regressive model stored in the video segments set207 (Eqs. (2) and (3)) can be expressed using Eq. (4):

$\begin{matrix}{\begin{bmatrix}x_{1} \\x_{2} \\\vdots \\x_{n - 1}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 0 & 0 & \ldots & 0 \\1 & 1 & 1 & 0 & \ldots & 0 \\\vdots & \vdots & \vdots & \; & \ldots & \vdots \\1 & 1 & 1 & 1 & \ldots & 1\end{bmatrix}\begin{bmatrix}x_{1} \\{\Delta \; x_{1}} \\\vdots \\{\Delta \; x_{n - 1}}\end{bmatrix}}} & (4)\end{matrix}$

where [X₁, X₂ , . . . , X_(n-1)]^(T) are known and [x₁, Δx₁, . . .Δx_(n-1)]^(T) are unknowns. Clearly, there are more unknowns than thenumber of equations. Furthermore, video frames corresponding to the samescene are expected to display smooth transitions. Therefore, vector [x₁, Δx₁, . . . Δx_(n-1)]^(T) is expected to be sparse (i.e., very fewelements of this vector should be non-zero). Therefore, in the preferredembodiment of the present invention, [x₁, Δx₁, . . . Δx_(n-1)]^(T) isestimated by applying sparse solver on Eq. (4). Mathematical algorithmsfor determining sparse combinations are well-known in the art. Anin-depth analysis of sparse combinations, their mathematical structureand relevancy can be found in the article entitled “From sparsesolutions of systems of equations to sparse modeling of signals andimages,” (SIAM Review, pp. 34-81, 2009) by Bruckstein et al., which isincorporated herein by reference.

Similarly, [y₁, Δy₁, . . . Δy_(n-1)]^(T) is estimated by solving thelinear equation given by Eq. (5) using a sparse solver:

$\begin{matrix}{\begin{bmatrix}Y_{1} \\Y_{2} \\\vdots \\Y_{n - 1}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 0 & 0 & \ldots & 0 \\1 & 1 & 1 & 0 & \ldots & 0 \\\vdots & \vdots & \; & \; & \ldots & \; \\1 & 1 & 1 & 1 & \ldots & 1\end{bmatrix}\begin{bmatrix}y_{1} \\{\Delta \; y_{1}} \\\vdots \\{\Delta \; y_{n - 1}}\end{bmatrix}}} & (5)\end{matrix}$

Note that, from Eqs. (2), and (3), it is clear that knowledge of [x₁,Δx₁, . . . Δx_(n-1)]^(T), and [y₁, Δy₁, . . . Δy_(n-1)]^(T) issufficient to determine x_(i), and y_(i), respectively, ∀i, 1≦i≦n. Theaffine transform coefficients set 211 is determined by collectingvectors [x₁, Δx₁, . . . Δx_(n-1)]^(T), and [y₁, Δy₁, . . .Δy_(n-1)]^(T).

FIG. 4 is a more detailed view of the get common and dynamic videoframes step 212 according to a preferred embodiment. In a define firstset of basis functions step 402, a set of basis functions that can beused to estimate a common scene content for the selected video segment209 is defined. The set of basis functions produced by the define firstset of basis functions step 402 is collected as first set of basisfunctions 403. In a preferred embodiment the first set of basisfunctions 403 are a set of DCT basis functions. DCT basis functions arewell-known in the art. For example, the article “K-SVD: An algorithm fordesigning overcomplete dictionaries for sparse representation” by Aharonet al. (IEEE Transactions on Signal Processing, Vol. 54, pp. 4311-4322,2006) defines a set of DCT basis functions that can be used inaccordance with the present invention. In other embodiments, other setsof basis functions can alternatively be used, such as a set of waveletbasis functions, a set of delta function basis functions or a set ofbasis functions determined by analyzing a set of training images.

A determine common video frame step 404 determines a common video frame405 in response to the first set of basis functions 403 as given by Eq.(6) below:

C=ψβ  (6)

where C is a vector representation of the common video frame 405 and ψis a matrix representation of the first set of basis functions 403. β isa sparse vector of weighting coefficients where only a minority of theelements of β are non-zero. The matrix ψ can be determined in anyappropriate way known to those skilled in the art. In a preferredembodiment, ψ is a discrete cosine transform (DCT) matrix.

In a define second set of basis functions step 406, a set of basisfunctions that can be used to estimate a set of dynamic scenes for theselected video segment 209 is defined. The set of basis functionsproduced by the define second set of basis functions step 406 iscollected as second set of basis functions 407. In a preferredembodiment, the second set of basis functions 407 is the same set of DCTbasis functions that were used for the first set of basis functions 403.However, in other embodiments a different set of basis functions can beused.

A determine dynamic video frames step 408 determines a dynamic videoframes set 409 responsive to the second set of basis functions 407. Thedynamic video frames set 409 can be determined in any appropriate wayknown to those skilled in the art. In a preferred embodiment, a set ofsparse linear combinations of the basis functions of the second set ofbasis functions 407 is determined to represent the dynamic video framesset 409 as given by Eq. (7) below:

D_(i)=φα_(i); 1≦i≦n   (7)

where D_(i) is the vector representation of the dynamic scenecorresponding to f_(i) and φ is the matrix representation of the secondset of basis functions 407, and α_(i)(1≦i≦n) are sparse vectors ofweighting coefficients. In a preferred embodiment, φ is assumed to besame as ψ (i.e., φ=ψ).

A determine common and dynamic video frames step 410 produces the commonand dynamic video frames set 213 responsive to the affine transformcoefficients set 211, the selected video segment 209, the common videoframe 405, and the dynamic video frames set 409. The common and dynamicvideo frames set 213 can be determined in any appropriate way known tothose skilled in the art. In a preferred embodiment, the determinecommon and dynamic video frames step 410 solves Eq. (8) to determine thecommon and dynamic video frames set 213.

$\begin{matrix}{\begin{bmatrix}f_{1} \\f_{2} \\f_{3} \\\vdots \\f_{n}\end{bmatrix} = {\begin{bmatrix}{{T\left( \Theta_{1} \right)}\psi} & \psi & 0 & \ldots & 0 \\{{T\left( \Theta_{2} \right)}\psi} & 0 & \psi & \ldots & 0 \\\vdots & \; & \; & \; & \; \\{{T\left( \Theta_{n} \right)}\psi} & 0 & 0 & \ldots & \psi\end{bmatrix}\begin{bmatrix}\beta \\\alpha_{1} \\\alpha_{2} \\\vdots \\\alpha_{n}\end{bmatrix}}} & (8)\end{matrix}$

From Eq. (8), it is clear that f_(i)=T(Θ_(i))C+D_(i), whereΘ_(i)={x_(i), y_(i)}, C=ψβ, and D_(i)=φα_(i)=ψα_(i). Due to the sparsenature of β and α_(i), vector [β, α₁, . . . , α_(n)]^(T) is estimatedusing a sparse solver. Mathematical algorithms to solve the linearequation of the form shown in Eq. (9) for determining sparse vector arewell-known in the art. An in-depth analysis of sparse solvers, theirmathematical structures and relevancies can be found in theaforementioned article by Bruckstein et al. entitled “From SparseSolutions of Systems of Equations to Sparse Modeling of Signals andImages.” The common and dynamic video frames set 213 is determined bycollecting the common video frame C and the dynamic video frames D_(i)(1≦i≦n), where C=ψβ, and D_(i)=ψα_(i).

FIG. 5 shows an example of a video segment 502 including five videoframes. A common video frame 504 and dynamic video frames 506corresponding to the video segment 502 determined using the method shownin FIG. 2 are also shown. It can be seen that the common scene contentin the video segment 502 is captured by the common video frame 504,while the variable scene content is captured by the dynamic video frames506.

The common and dynamic video frames set 213, in conjunction with theaffine transform coefficients set 211, contain sufficient information toreconstruct the selected video segment 209. FIG. 6 illustrates theformation of a reconstructed video segment set 603 according to apreferred embodiment. A reconstruct video segment step 602 uses thecommon and dynamic video frames set 213 and the affine transformcoefficients set 211 to form the reconstructed video segment set 603,which represents an estimate of the selected video segment 209. Thereconstructed video segment set 603 can be determined in any appropriateway known to those skilled in the art. In a preferred embodiment, thereconstruct video segment step 602 uses Eq. (9) to reconstruct theselected video segment 209:

{circumflex over (f)}_(i) =T(Θ_(i))C+D _(i)   (9)

where {circumflex over (f)}_(i) is the reconstructed estimate of thei^(th) video frame, f_(i), of the selected video segment 209. Thereconstructed video frames {circumflex over (f)}_(i) (1≦i≦n) arecollected in the reconstructed video segment set 603. Due to the noiserobustness property of sparse solvers, the reconstructed video segmentset 603 is robust to noise. In other words, denoising is automaticallyachieved during the video reconstruction process.

FIG. 7 shows an example of a noisy video segment 702. A common sceneimage 704 and dynamic scene images 706 corresponding to the noisy videosegment 702 were determined according to the method of FIG. 2. Areconstructed denoised video 708 is also shown, which was determinedaccording to the method shown in FIG. 6. This example clearlyillustrates the denoising property of the algorithm described here.

In addition to reconstruction and denoising, the proposed algorithm canbe used for many useful video editing and tracking applications withoutperforming motion estimation and compensation. A preferred embodiment ofa method for modifying the common scene content of the selected videosegment 209 is shown in FIG. 8. An extract dynamic video frames step 802extracts the dynamic video frames (D₁, D₂ , . . . , D_(n)) from thecommon and dynamic video frames set 213 to provide a dynamic videoframes set 409. A determine new common video frame step 804 provides anew common video frame 805 that is used to modify the common scenecontent of the selected video segment 209. In a preferred embodiment, auser interface is provided enabling a user to manually select the newcommon video frame 805 according to user preference. A reconstruct videosegment step 806 uses the dynamic video frames set 803 and the newcommon video frame 805 to produce the reconstructed video segment set807. The video frames of the reconstructed video segment set 807 inheritthe dynamic scene contents from the selected video segment 209, but havedifferent common scene content as explained next.

The reconstructed video segment set 807 can be determined in anyappropriate way known to those skilled in the art. In a preferredembodiment, the reconstruct video segment step 806 uses Eq. (10) toproduce the reconstructed video segment set 807:

f _(i) ^(R) =νC ^(N) +ρD _(i)   (10)

where f_(i) ^(R) is the reconstructed version of the i^(th) video frame,f_(i), of the selected video segment 209, C^(N) is the value of the newcommon video frame 805, and ν and ρ are constants. In a preferredembodiment, ν and ρ are pre-determined constants that control the visualquality of f_(i) ^(r). The reconstructed video frames f_(i) ^(R) (1≦i≦n)are collected in the reconstructed video segment set 807.

Similar to the application described in FIG. 8 where the common scenecontent of the selected video segment 209 is replaced with new commonscene content, in some embodiments, the dynamic scene content in thedynamic video frames can be replaced with new dynamic scene content andcombined with the original common scene content to provide a newreconstructed video segment set 807.

FIG. 9 illustrates a method for detecting moving objects in the selectedvideo segment 209 in accordance with the present invention. An extractdynamic video frames step 902 extracts the dynamic video frames (D₁, D₂,. . . , D_(n)) from the common and dynamic video frames set 213 toprovide a dynamic video frames set 903. A detect moving objects step 904determined the co-ordinates of the moving objects present in theselected video segment 209 responsive to the dynamic video frames set903. The co-ordinates of the moving objects produced by the detectmoving objects step 904 are stored in a moving objects set 905. Themoving objects set 905 can be determined in any appropriate way known tothose skilled in the art. In a preferred embodiment, the detect movingobjects thresholds the pixel values of the dynamic video frames D₁, D₂,. . . , D_(n) as shown in Eq. (11):

$\begin{matrix}{{D_{i}\left( {r,s} \right)} = \left\{ \begin{matrix}1 & {{{if}\mspace{14mu} {{D_{i}\left( {r,s} \right)}}} > T} \\0 & {{otherwise},}\end{matrix} \right.} & (11) \\{1 \leq i \leq n} & \;\end{matrix}$

where T is a threshold. The threshold T can be determined in anyappropriate way known to those skilled in the art. In some embodiments,the threshold T is a predetermined constant. However, it has been foundthat in many cases it is preferable for the threshold T to be videodependent. A user interface can be provided enabling the user to specifya heuristically determined threshold T that works best for a particularselected video segment 209. The co-ordinates corresponding to|D_(i)(r,s)|=1 are collected in the moving objects set 905.

The method described earlier with respect to FIGS. 3 and 4 forestimating the common and dynamic video frames assumes that the selectedvideo segment 209 contains only one scene including common scenecontent. However, in practice, an input digital video 203 may containmultiple scenes; therefore, it is desirable to detect scene boundariesautomatically. After determining scene boundaries, a set of common anddynamic video frames can be estimated for the individual video segmentscorresponding to each scene in accordance with the method of the presentinvention described above. A scene boundary detection method thatexploits the algorithm presented above to automatically detect the sceneboundaries in a video is presented next.

FIG. 10 is a flow diagram illustrating a method for determining a sceneboundary between a first scene and a second scene in the intermediatedigital video 205 including a time sequence of input video frames,according to an embodiment of the present invention. In a preferredembodiment, the intermediate digital video 205 is divided into aplurality of digital video sections 1003 and each of the digital videosections 1003 is analyzed to determine whether it contains a sceneboundary. The time duration of the digital video sections 1003 is chosento be small enough that it is unlikely that they would contain more thanone scene boundary (e.g, 10 video frames). An extract digital videosection step 1002 extracts a particular digital video section 1003 fromthe intermediate digital video 205 for analysis. In a preferredembodiment, the set of digital video sections 1003 are defined such thatconsecutive digital video sections 1003 overlap slightly in order toavoid missing scene boundaries that happen to occur at the end of adigital video section 1003.

FIG. 11 shows a diagram illustrating an intermediate digital video 205that includes three video segments 209 corresponding to differentscenes, which are divided by scene boundaries at scene boundarylocations 1015. The intermediate digital video 205 is divided into a setof M overlapping digital video sections 1003 (V₁−V_(M)). In accordancewith the present invention, the method of FIG. 10 is applied to each ofthe digital video sections 1003 to determine whether they contain ascene boundary, and if so to determine the scene boundary location 1015.

Returning to a discussion of FIG. 10, a define set of basis functionsstep 1004 defines a set of basis functions for representing the dynamicscene content of the digital video section 1003. The set of basisfunctions is collected in a basis functions set 1005. In a preferredembodiment, the basis functions set 1005 is the same set of DCT basisfunctions that were discussed earlier with respect to FIG. 4.

An evaluate merit function step 1006 evaluates a merit function for aset of candidate scene boundary locations 1007. The evaluate meritfunction step 1006 analyzes the digital video section 1003 responsive tothe basis functions set 1005 for each of the candidate scene boundarylocations 1007 to determine corresponding merit function values 1009.The merit function values 1009 provide an indication of the likelihoodthat a particular candidate scene boundary location 1007 corresponds toa scene boundary. A preferred form for the merit function will bedescribed relative to FIG. 13, but any appropriate merit function can beused in accordance with the present invention.

A scene boundary present test 1010 evaluates the determined meritfunction values 1009 to determine whether a scene boundary is present inthe digital video section 1003. Let S={π₁, π₂, . . . , π_(ω)} be thecandidate scene boundary location 1007, wherein each π_(i) ∈ [1, . . . ,N], 1≦i≦ω. The corresponding set of merit function values 1009 can berepresented as Π={MF_(π) ₁ , MF_(π) ₂ , . . . , MF_(π) _(ω) )}. In apreferred embodiment of the present invention, the scene boundarypresent test 1010 determines the maximum merit function valueΠ_(max)=max(Π) and the minimum merit function value Π_(min)=min(Π) inthe set of merit function values 1009. The scene boundary present test1010 determines that a scene boundary is present if a ratio betweenΠ_(max) and Π_(min) is less than a predefined threshold. That is, thedigital video section 1003 is designated to have a scene boundary ifΠ_(max)/Π_(min)≧T_(s), where T_(s) is a predefined threshold.

FIG. 12A shows a graph 1050 plotting the merit function value 1009 (MF)as a function of the candidate scene boundary location 1007 (π) for adigital video section 1003 that includes a scene boundary. Likewise,FIG. 12B shows a graph 1052 plotting the merit function value 1009 (MF)as a function of the candidate scene boundary location 1007 (π) for adigital video section 1003 that does not include a scene boundary. Itcan be seen that the range between Π_(max) and Π_(min) is much smallerin FIG. 12B than it was for FIG. 12A.

If the scene boundary present test 1010 determines that no sceneboundary is present (i.e., Π_(max)/Π_(min)<T_(s)), then a no sceneboundary found step 1012 is used to indicate that the digital videosection 1003 does not include a scene boundary.

If the scene boundary present test 1010 determines that a scene boundaryis present, then a determine scene boundary location step 1014determines a scene boundary location 1015 which divides the digitalvideo section 1003 into first and second scenes responsive to the meritfunction values 1009.

In a preferred embodiment, the scene boundary location 1015 is definedto be the candidate scene boundary location 1007 corresponding to theminimum merit function value in the set of merit function values 1009.The determine scene boundary location step 1014 selects π_(min), whichis the element of S that corresponds to the minimum merit function valueΠ_(min)=Min(Π)=MF_(π) _(min) , to be the scene boundary location 1015.This is illustrated in FIG. 12A, which shows the designated sceneboundary location 1015 that corresponds to the position of the minimummerit function value Π_(min) that occurs at π_(min).

The discussion above describes the case where the candidate sceneboundary locations 1007 includes each of the video frames in the digitalvideo section 1003. This corresponds to performing an exhaustive searchof all of the possible candidate scene boundary locations. One skilledin the art will recognize that other embodiments can use other searchtechniques to identify the candidate scene boundary location 1007producing the minimum merit function value Π_(min). For example, aniterative search technique, such as the well-known golden section searchtechnique, can be used to converge on the desired solution for the sceneboundary location 1015. Such iterative search techniques have theadvantage that they require fewer computations as compared to theexhaustive search technique.

The method discussed relative to FIG. 10 is repeated for each of thedigital video sections 1003 that were defined from the intermediatedigital video 205. The get video segments step 206 of FIG. 2 then usesthe resulting set of scene boundary locations 1015 to segment theintermediate digital video 205 into the video segments set 207.

The evaluate merit function step 1006 evaluates a predefined meritfunction for each of the candidate scene boundary locations 1007 todetermine corresponding merit function values 1009. FIG. 13 is a flowchart illustrating the computation of a merit function value 1009 for aparticular candidate scene boundary location 1007 according to apreferred embodiment of the present invention.

The set of candidate scene boundary locations 1007 that are evaluatedcan be determined using any method known to those skilled in the art. Ina preferred embodiment of the present invention, each of the videoframes in the digital video section 1003 are evaluated as a candidatescene boundary location 1007. Let ζ₁, ζ₂, . . . , ζ_(N) be the videoframes stored in the digital video section 1003. Let π be the value ofthe candidate scene boundary location 1007, then π∈ {1, 2, . . . ,N}where N is the total number of video frames in the digital videosection 1003.

A determine left and right video frames sets step 1104 partitions thedigital video section 1003 into a left video frames set 1105 and a rightvideo frames set 1107 by dividing the digital video section 1003 at thecandidate scene boundary location 1007 (π). Accordingly, the left videoframes set 1105 contains the video frames of the digital video section1003 preceding the candidate scene boundary location 1007 (i.e., ζ₁, ζ₂,. . . , ζ_(π−1)). Similarly, the right video frames set 1107 containsthe video frames following the candidate scene boundary location 1007(i.e., ζ_(π), ζ_(π+1), . . . , ζ_(N)).

A get left dynamic content step 1108 uses the basis functions set 1005to determine left dynamic content 1109 providing an indication of thedynamic scene content in the left video frames set 1105. In a preferredembodiment, the dynamic scene content for each of the video frames ζ₁,ζ₂, . . . , ζ_(π−1) in the left video frames set 1105 is representedusing a sparse combination of the basis functions in the basis functionsset 1005, wherein the sparse combination of the basis functions isdetermined by finding a sparse vector of weighting coefficients for eachof the basis function in the basis functions set 1005. The sparse vectorof weighting coefficients for each video frame in the left video framesset 1105 can be determined using any method known to those skilled inthe art. In a preferred embodiment, the same method that was discussedrelative to FIG. 2 is used to estimate the common and the dynamic scenecontents for the video frames ζ₁, ζ₂, . . . ζ_(π−1) in the left videoframes set 1105. Accordingly, the basis functions of the basis functionsset 1005 are used in Eq. (8) to estimate the common scene content(C^(L)) and the dynamic scene content (D^(L) ₁, . . . , D_(π−1) ^(L))for the video frames ζ₁, ζ₂, . . . , ζ_(π−1), where D_(τ) ^(L)=λα_(τ)^(L); 1≦τ≦π-1, λ is the matrix representation of the basis functions inthe basis functions set 1005 and α_(τ) ^(L) is a sparse vector ofweighting coefficients corresponding to the τ^(th) dynamic content. Theresulting sparse vector of weighting coefficients (α₁ ^(L), α₂ ^(L), ,α_(π−1) ^(L)) is stored as the left dynamic content 1109.

Similarly, a get right dynamic content step 1110 uses the basisfunctions set 1005 to determine right dynamic content 1111 providing anindication of the dynamic scene content in the right video frames set1107. In a preferred embodiment, the dynamic scene content for each ofthe video frames ζ_(π), ζ_(π+1), . . . , ζ_(N) in the right video framesset 1107 is represented using a sparse combination of the basisfunctions in the basis functions set 1005, wherein the sparsecombination of the basis functions is determined by finding a sparsevector of weighting coefficients for each of the basis function in thebasis functions set 1005. The sparse vector of weighting coefficientsfor each video frame in the right video frames set 1107 can bedetermined using any method known to those skilled in the art. In apreferred embodiment, the method that was discussed relative to FIG. 2is used to estimate the common and the dynamic scene contents for thevideo frames ζ_(π), ζ_(π+1), . . . , ζ_(N) in the right video frames set1107. Accordingly, the basis functions of the basis functions set 1005are used in Eq. (8) to estimate the common scene content (C^(R)) and thedynamic scene content (D_(π) ^(L), . . . , D_(N)) for the frames ζ_(π),ζ_(π+1), . . . , ζ_(N), where D_(τ) ^(R)=λα_(τ) ^(R); π≦τ≦N, λ is thematrix representation of the basis functions in the basis functions set1005 and α_(τ) ^(R) is a sparse vector of weighting coefficientscorresponding to the τ^(th) dynamic content. The resulting sparse vectorof weighting coefficients (α_(π) ^(R), α_(π+1) ^(R), ,α_(N) ^(R)) isstored as the right dynamic content 1111.

A compute merit function value step 1112 determines the merit functionvalue 1009 by combining the left dynamic content 1109 and the rightdynamic content 1111. The compute merit function value step 1112 can useany method known to those skilled in the art to determine the meritfunction value 1009. In a preferred embodiment, the weightingcoefficients in the left dynamic content 1109 and the right dynamiccontent 1111 are concatenated to form a combined vector of weightingcoefficients. The compute merit function value step 1112 the computes anl-1 norm of the combined vector of weighting coefficients to determinethe merit function value 1009 as given by Eq. (12):

MF_(π)=∥[α₁ ^(L), . . . , α_(π−1) ^(L), α_(π) ^(R), . . . , α_(N)^(R)]^(T)∥₁   (12)

where MF_(π) is the merit function value 1009 for the candidate sceneboundary location 1007 (π), and ∥•∥₁ denotes l-1 norm.

In a preferred embodiment, the get video segments step 206 of FIG. 2uses the set of scene boundary locations 1015 determined using themethod of FIG. 10 to segment the intermediate digital video 205 into thevideo segments set 207. Each video segment corresponds to the sequenceof video frames extending from one scene boundary location 1015 to thenext.

It is to be understood that the exemplary embodiments disclosed hereinare merely illustrative of the present invention and that manyvariations of the above-described embodiments can be devised by oneskilled in the art without departing from the scope of the invention. Itis therefore intended that all such variations be included within thescope of the following claims and their equivalents.

PARTS LIST

-   110 data processing system-   120 peripheral system-   130 user interface system-   140 data storage system-   202 receive input digital video step-   203 input digital video-   204 initialize intermediate digital video step-   205 intermediate digital video-   206 get video segments step-   207 video segments set-   208 select video segment step-   209 video segment-   210 get affine transform coefficients step-   211 affine transform coefficients set-   212 get common and dynamic video frames step-   213 common and dynamic video frames set-   302 determine transform coefficients model step-   303 transform coefficients model set-   304 determine measurement vector step-   305 measurement vector set-   306 estimate affine transform coefficients step-   402 define first set of basis functions step-   403 first set of basis functions-   404 determine common video frame step-   405 common video frame-   406 define second set of basis functions step-   407 second set of basis functions-   408 determine dynamic video frames step-   409 dynamic video frames set-   410 determine common and dynamic video frames step-   502 video segment-   504 common video frame-   506 dynamic video frames-   602 reconstruct video segment step-   603 reconstructed video segment set-   702 noisy video segment-   704 common scene image-   706 dynamic scene images-   708 reconstructed video segment-   802 extract dynamic video frames step-   803 dynamic video frames set-   804 determine new common video frame step-   805 new common video frame-   806 reconstruct video segment step-   807 reconstructed video segment set-   902 extract dynamic video frames step-   903 dynamic video frames set-   904 detect moving objects step-   905 moving objects set-   1002 extract digital video section step-   1003 digital video section-   1004 define set of basis functions step-   1005 basis functions set-   1006 evaluate merit function step-   1007 candidate scene boundary locations-   1009 merit function values-   1010 scene boundary present test-   1012 no scene boundary found step-   1014 determine scene boundary location step-   1015 scene boundary location-   1104 determine left and right video frames sets step-   1105 left video frames set-   1107 right video frames set-   1108 get left dynamic content step-   1109 left dynamic content-   1110 get right dynamic content step-   1111 right dynamic content-   1112 compute merit function value step

1. A method for representing a video sequence including a time sequenceof input video frames, the input video frames including some commonscene content that is common to all of the input video frames and somedynamic scene content that changes between at least some of the inputvideo frames, comprising: a) determining an affine transform having aset of affine transform coefficients for each input video frame that canbe used to align the common scene content in the input video frames; b)defining a first set of basis functions for representing the commonscene content; c) defining a second set of basis functions forrepresenting the dynamic scene content; d) automatically determining acommon video frame including the common scene content by forming asparse combination of the first basis functions; e) automaticallydetermining a dynamic video frame for each input video frame by forminga sparse combination of the second basis functions, wherein the dynamicvideo frames can be combined with the respective affine transforms andthe common video frame to provide reconstructed video frames providingrepresentations of the respective input video frames; and f) storingrepresentations of the common video frame and the dynamic video framesin a processor-accessible memory; wherein the method is performed atleast in part using a data processing system.
 2. The method of claim 1wherein the common video frame C is determined using the equation:C=ψβ and the i^(th) dynamic video frame D_(i) is determined using theequation:D_(i)=φα_(i). where ψ is the first set of basis functions, φ is thesecond set of basis functions, β is a sparse set of weightingcoefficients for representing the common video frame, α_(i) is a sparseset of weighting coefficients for representing the i^(th) dynamic videoframes, and wherein β and α_(i) are determined by solving the equation:f _(i) =T(Θ_(i))ψβ+φα_(i) where f_(i) is the i^(th) input video frames,T(Θ_(i)) is an affine transform based on the affine transformcoefficients Θ_(i) for the i^(th) frame.
 3. The method of claim 1wherein at least one of the sets of basis functions is a set of DCTbasis functions, a set of wavelet basis functions or a set of deltafunction basis functions.
 4. The method of claim 1 wherein the first setof basis functions is the same as the second set of basis functions. 5.The method of claim 1 wherein the affine transform coefficients arerepresented using an auto regressive model.
 6. The method of claim 1wherein the affine transform coefficients are determined using a sparserepresentation framework.
 7. The method of claim 1 further includingdetermining a reconstructed video sequence including a time sequence ofreconstructed video frames, wherein each reconstructed video frame isdetermined by combining the common video frame with the respectivedynamic video frames and respective affine transforms.
 8. The method ofclaim 7 wherein the reconstructed video sequence has a lower noise levelthan the input video sequence.
 9. The method of claim 1 furtherincluding determining a modified video sequence including a timesequence of modified video frames, wherein each modified video frame isdetermined by combining the respective dynamic video frames with a newcommon video frame.
 10. The method of claim 9 wherein the modified videoframes are determined by computing a weighted combination of therespective dynamic video frame and the new common video frame.
 11. Themethod of claim 1 further including determining a modified videosequence including a time sequence of modified video frames, whereineach modified video frame is determined by combining a corresponding newdynamic video frame from a set of new dynamic video frames with thecommon video frame.
 12. The method of claim 1 further includingdetecting moving objects in the video sequence by analyzing the dynamicvideo frames.
 13. The method of claim 12 wherein the dynamic videoframes are analyzed by applying a threshold operation to detect pixelscorresponding to moving objects.